Logan County
Evaluating Self-Generated Documents for Enhancing Retrieval-Augmented Generation with Large Language Models
Li, Jiatao, Hu, Xinyu, Yin, Xunjian, Wan, Xiaojun
The integration of documents generated by LLMs themselves (Self-Docs) alongside retrieved documents has emerged as a promising strategy for retrieval-augmented generation systems. However, previous research primarily focuses on optimizing the use of Self-Docs, with their inherent properties remaining underexplored. To bridge this gap, we first investigate the overall effectiveness of Self-Docs, identifying key factors that shape their contribution to RAG performance (RQ1). Building on these insights, we develop a taxonomy grounded in Systemic Functional Linguistics to compare the influence of various Self-Docs categories (RQ2) and explore strategies for combining them with external sources (RQ3). Our findings reveal which types of Self-Docs are most beneficial and offer practical guidelines for leveraging them to achieve significant improvements in knowledge-intensive question answering tasks.
- Asia > Russia (1.00)
- Europe > Poland > Masovia Province > Warsaw (0.04)
- Europe > Eastern Europe (0.04)
- (14 more...)
- Transportation > Air (1.00)
- Media > Film (1.00)
- Leisure & Entertainment (1.00)
- (5 more...)
Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework
Chen, Lu, Zhang, Ruqing, Guo, Jiafeng, Fan, Yixing, Cheng, Xueqi
Retrieval-augmented generation (RAG) has emerged as a popular solution to mitigate the hallucination issues of large language models. However, existing studies on RAG seldom address the issue of predictive uncertainty, i.e., how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications. In this work, we emphasize the importance of risk control, ensuring that RAG models proactively refuse to answer questions with low confidence. Our research identifies two critical latent factors affecting RAG's confidence in its predictions: the quality of the retrieved results and the manner in which these results are utilized. To guide RAG models in assessing their own confidence based on these two latent factors, we develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers. We also introduce a benchmarking procedure to collect answers with the option to abstain, facilitating a series of experiments. For evaluation, we introduce several risk-related metrics and the experimental results demonstrate the effectiveness of our approach. Our code and benchmark dataset are available at https://github.com/ict-bigdatalab/RC-RAG.
- Asia > China > Beijing > Beijing (0.04)
- North America > United States > Kentucky > Logan County (0.04)
- Europe > United Kingdom > England > Cambridgeshire > Cambridge (0.04)
- Asia > Middle East > Jordan (0.04)